Anomaly Detection in Time Series

Tensorflow version: 2.1.0

S&P 500 Index Data

Data Source: S&P500 Daily Prices 1986 - 2018

	date	close
0	1986-01-02	209.59
1	1986-01-03	210.88
2	1986-01-06	210.65
3	1986-01-07	213.80
4	1986-01-08	207.97

(8192, 2)

Data Preprocessing

(6553, 2) (1639, 2)

Training and Test Splits

(6523, 30, 1)

LSTM Autoencoder

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm (LSTM)                  (None, 128)               66560     
_________________________________________________________________
dropout (Dropout)            (None, 128)               0         
_________________________________________________________________
repeat_vector (RepeatVector) (None, 30, 128)           0         
_________________________________________________________________
lstm_1 (LSTM)                (None, 30, 128)           131584    
_________________________________________________________________
dropout_1 (Dropout)          (None, 30, 128)           0         
_________________________________________________________________
time_distributed (TimeDistri (None, 30, 1)             129       
=================================================================
Total params: 198,273
Trainable params: 198,273
Non-trainable params: 0
_________________________________________________________________

Training the Autoencoder

Train on 5870 samples, validate on 653 samples
Epoch 1/100
5870/5870 [==============================] - 12s 2ms/sample - loss: 0.1625 - val_loss: 0.1610
Epoch 2/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.1114 - val_loss: 0.0986
Epoch 3/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0903 - val_loss: 0.0443
Epoch 4/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0802 - val_loss: 0.0442
Epoch 5/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0717 - val_loss: 0.0639
Epoch 6/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0776 - val_loss: 0.0327
Epoch 7/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0750 - val_loss: 0.0313
Epoch 8/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0744 - val_loss: 0.0578
Epoch 9/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0758 - val_loss: 0.0522
Epoch 10/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0765 - val_loss: 0.0302
Epoch 11/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0734 - val_loss: 0.0610
Epoch 12/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0749 - val_loss: 0.0629
Epoch 13/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0751 - val_loss: 0.0274
Epoch 14/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0746 - val_loss: 0.0407
Epoch 15/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0741 - val_loss: 0.0454
Epoch 16/100
5870/5870 [==============================] - 7s 1ms/sample - loss: 0.0762 - val_loss: 0.0277

Plotting Metrics and Evaluating the Model

1609/1609 [==============================] - 1s 374us/sample - loss: 0.2685

0.26845344528787396

Detecting Anomalies in the S&P 500 Index Data

	date	close	loss	threshold	anomaly
8187	2018-06-25	4.493228	0.638412	0.65	False
8188	2018-06-26	4.507583	0.691408	0.65	True
8189	2018-06-27	4.451431	0.696458	0.65	True
8190	2018-06-28	4.491406	0.727353	0.65	True
8191	2018-06-29	4.496343	0.709381	0.65	True

	date	close	loss	threshold	anomaly
7474	2015-08-25	2.457439	0.655041	0.65	True
7475	2015-08-26	2.632149	0.711078	0.65	True
8090	2018-02-05	4.329949	0.657327	0.65	True
8091	2018-02-06	4.440671	0.846347	0.65	True
8092	2018-02-07	4.408365	0.822247	0.65	True

	date	close	loss	threshold	anomaly
8145	2018-04-25	4.307086	0.653256	0.65	True
8188	2018-06-26	4.507583	0.691408	0.65	True
8189	2018-06-27	4.451431	0.696458	0.65	True
8190	2018-06-28	4.491406	0.727353	0.65	True
8191	2018-06-29	4.496343	0.709381	0.65	True